zYolaaliate mele (-vialole 


Adapting methods for family and 
community research during a pandemic 


With contributions from: 
Angela Calabrese Barton 
meclarel xeon =I eM Or-lant-Vouale) 
Elizabeth A. Davis 
Akeiylah DeWitt 

Alexis Hiniker 

SIU (om Cre] cel) 

Day Greenberg 


Lori Takeuchi 
(orVidl fan em eladn 
Brigid Barron 


Leslie Rupert Herrenkohl 
Denise Jones 
Veronica Joyce Lin 


Spring 2021 Julie Kientz 


Cindy K. Lam 
Zohar Levy 
Rebecca Michelson 
Sean A. Munson 
Judy Nguyen 
Rose K. Pozos 
Devon Riter 
Peter Siciliano 
Tammy Tasker 
Chandler Turner 
Jason Yip 


The Joan Ganz Cooney Center at Sesame Workshop 


This report is based upon work supported by the National Science Foundation under 
Grant No. 2028082. Additionally, the report describes research funded by Grant Nos. 
2028370 and 2027525. Any opinions, findings, and conclusions or recommendations 
expressed in this material are those of the authors and do not necessarily reflect the 
views of the National Science Foundation. 


ABOUT THE AUTHORS 


Lori Takeuchi, PhD is a program director at the National Science Foundation in 

its Division of Research on Learning. Prior to joining NSF, she spent 12 years at the 

Joan Ganz Cooney Center at Sesame Workshop. A learning scientist by training, 

her research focuses on how children use media and technology across the various 
settings of their lives and the implications these tools hold for their learning and 
development. Before earning her PhD from Stanford’s Graduate School of Education, 
Lori spent over a decade designing curriculum-based science simulation and visualization 
software. She began her career managing Instructional Television at Thirteen/WNET. 


Caitlin K. Martin is a consultant with over 20 years of experience researching and 
designing learning environments, with a focus on out-of-school opportunities, youth 
fo}coXe[UCad Lola iidamc-Lelal ale) Coy:AeMel-le-MVAKIUL-1IP-lalolape-lalemeolanlane lain all aicelanatsteme(-ii-aam-Vale| 
interpretation. She regularly collaborates with Brigid Barron’s research lab at Stanford 
University, The Office of Community Education Partnerships (OCEP) at Northwestern 
UTaTNZlasiin vere lale im K-Vel alate) (oy:aVmie) msioteli-]@ Cfeorote =] oll UES\CH Mr-] 0))k-] a BY-1-7-10] MOla TI -TESIINVA 

Caitlin has also conducted evaluations of regional and national project including the 
American Library Association’s Libraries Ready to Code initiative. 


Brigid Barron, PHD is a Professor of Education and the Learning Sciences at Stanford’s 
Graduate School of Education. Her research investigates the dynamics of learning 
ecologies with a focus on how digital technologies can serve as catalysts for collaborative 
and interest learning at home, school, and in the community with the goal of creating more 
equitable opportunities for the development of expertise and interest-driven learning. 


A full-text PDF of this publication is available as a free download from 
www.joanganzcooneycenter.org. 

SUGGESTED CITATION 

Takeuchi, L., Martin, C. K., & Barron, B. (2021). Learning together: Adapting methods for 
family and community research during a pandemic. New York: Joan Ganz Cooney Center 


at Sesame Workshop. 


Learning together: Adapting methods for family and community research during a pandemic 
is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. 


Contents 


p~) 


13 


Lf 


22 
24 


29 


29 


30 


33 


34 
34 


34 


37 


38 


BACKGROUND 
About the Remote Methods Workshop 


CASE STUDIES OF REMOTE RESEARCH 

STUDY 1 Using remote diary methods to understand how 
families navigate emergency homeschooling driven 
by COVID-19 - Stanford University 

STUDY 2 Education, work, and life during COVID-19: 
Supporting families at home with technology - 
University of Washington 

STUDY 3 Attending to political and ethical dimensions of 
remote research methods — University of Michigan 


GUIDE TO REMOTE METHODS 
The opportunities and challenges of remote research methods 


BUILDING CAPACITY FOR REMOTE LEARNING: 
FUTURE RESEARCH AND DESIGN PRIORITIES 

Build collective understanding of findings from COVID-I9 
research on learning and education 


Strengthen the field’s capacity for rapid, collaborative, 
and useful remote research 


CLOSING THOUGHTS 


APPENDIX 
More about the Stanford study 


More about the University of Washington study 
More about the University of Michigan study 


REFERENCES 


pyele.<elgelulare 


Yo} Msi[aler=m atom ailek-ray4em eXelare(=1anl (ome lmhatsmarels 
the K-I2 American school system shut down 
(ok =Jahilc=)h ela Se] 0) 40) o)i\Vmomimel(emlamielcas 
yA0yA0 Pa =To Viale Maal ielasmonmsyiere(-lalcmelaremiareiie 
families to figure out how to do school at 
home. And while COVID-19 emerged well 
Ta Com iat=Mlavcelanate lite) amele(-Mmelam=1comaateld <-lom e)Y 
ijal=Mel=\(=1(0) o)aat=) a) mona c=relal ave) cere [ae] mrele) Mm LaTe 
have revolutionized the way we work, 
communicate, and learn, we still found 
ourselves tragically unprepared to carry on 
W/o} mo) aCeX=W ale] 0) e\=1al=1e Mla Ca Waele ls] colelaalcmel are) 
on college campuses. The spring shutdown 
sent teachers scrambling to deliver their 
ons entirely online, too often to students 
Toy (ore <-Yo M=)hiat=\muiaromare con el k-me) mm ofelate hw a(olie 
(-Yolaliavevaelliva ele] ailell eLei(oM lamiarostom lose ace 


The transition was challenging for all, if not devastating for 
students from populations disproportionately affected by 
COVID-19, including Black, Latinx, and lower-income families. 
Researchers across a variety of disciplines recognized the 
urgent need for understanding how this global “experiment” 

in home schooling is altering student learning, as well as 
students’ physical and emotional health, family dynamics, 

and teacher preparation and support, among many other 
things. And private, corporate, and government funders, in turn, 
responded by investing in such research as a means of ensuring 
the success and prosperity of an entire generation of students 
affected by the pandemic, acknowledging the need to reevaluate 
and reimagine formal education as we know it today. 


In response to the COVID-19 pandemic, the National Science 
Foundation (NSF) mobilized funding from its FY2020 budget 
and supplemental appropriations through the Coronavirus Aid, 
Relief, and Economic Security (CARES) Act. On March 5, 2020, 
NSF issued a Dear Colleague Letter (NSF 20-052) announcing 
its acceptance of proposals “to non-medical, non-clinical-care 
research that can be used immediately to explore how to model 
and understand the spread of COVID-19, to inform and educate 
about the science of virus transmission and prevention, and to 
encourage the development of processes and actions to address 
this global challenge” (National Science Foundation, 2020). 
Interested investigators were encouraged to apply through the 
NSF's Rapid Response Research (RAPID) funding mechanism, 
“a fast-tracked grant process to accelerate critical discoveries’ 
(National Science Foundation, 2021). 


Investigators could request up to $200,000 and one year to 
complete their research. Hundreds of researchers responded 
to the call for COVID-19 related proposals, and as of January 19, 
2021, NSF made 801 separate awards amounting to $208,132,911 
across six directorates (National Science Foundation, 2021). No 
fewer than 100 of these awards funded research that aimed to 
understand the pandemic’s impact on learning and education! 


Several large-scale surveys, including two funded by NSF's 
RAPID COVID-19 program, have documented that students and 
parents from low-income and marginalized populations are 
particularly strained by online learning (Aguilar et al., 2020; 
ParentsTogether, 2020; Pew Research Center, 2020; The Education 
Trust-West, 2020; University of Oregon CTN, 2020). Dr. Anna 
Rosefsky Saavedra and team from the University of Southern 
California Dornsife Center for Economic and Social Research 
(NSF #2037179), for instance, documented—through their 
nationally representative online survey of 1,400 U.S. families— 
that just two-thirds of low-income K-12 students (HH income 

< $25K) had access to laptop/desktop computers and/or the 
internet at home shortly after U.S. schools closed in April 2020, 
compared to more than 90% of students from middle and 
upper-income families (HH > $50K) (Polikoff et al., 2020). Drs. 
Jennifer Hamilton and Debbie Kim of NORC (NSF #2030436) 
employed both a nationally representative sample of 2,036 
students ages 13-17 and a learning management system serving 
approximately 2.5 million high school students nationwide, 


1 COVID Information Commons (CIC) provides a searchable database of all COVID-related studies funded by the 
National Science Foundation at https://covidinfocommons.datascience.columbia.edu/). CIC aims to facilitate 
knowledge sharing and collaboration across all NSF-funded COVID research efforts (NSF #2028999). 
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studying the extent to which the COVID-19 pandemic may be 
further narrowing the STEM pipeline for high school students 
with inadequate technological resources at home (NORC, 2020). 
Findings from these and other studies (e.g., Kuhfeld et al., 2020) 
may help quantify the magnitude and variability of the formal 
educational experiences of U.S. students during the pandemic 
and, in doing so, inform important policy decisions around 
school closures/reopenings and the provisioning of technology to 
support distance learning, among many other things. However, 
finer-grained, qualitative approaches are often needed for 
interpreting broad trends revealed through survey research 
and illuminating solutions for mitigating disparities between 
less- and more-advantaged families. These approaches may 
also reveal what’s been working for diverse families in their 
quarantined learning practices, thus providing insights that 
may result in more effective and equitable practices within 
formal education settings. 


Unfortunately, conducting both ethnographic and controlled 
studies as researchers once did—in the same physical space as 
the individuals under study—poses risks to participants and 
researchers alike for contracting the coronavirus. Researchers 
have consequently adjusted their methods in order to safely 
establish the higher-touch and sustained relationships with 
participants necessary to answer the how and why questions 
that surveys can fall short of addressing. NSF’s RAPID program 
has invested in several projects involving the use of remote data 
collection methods to study learning as it has been naturally 
unfolding during the pandemic. This includes our project—led 
by Dr. Brigid Barron at Stanford University’s Graduate School 


of Education and awarded by NSF's Science of Learning and 
Augmented Intelligence Program (NSF #2028082)—that proposed 
not only researching how families are adapting to school 


and policymakers concerned with achieving more equitable 
educational outcomes during and beyond the 

pandemic lockdowns. Specifically, our report aims to: 
shutdowns, but also mobilizing a broader community of 
investigators who are exploring innovative methods for studying 
home learning during the pandemic. To launch this community, 
Dr. Barron’s team, along with partners at the Joan Ganz Cooney 
Center, gathered two dozen researchers from the academic, 
nonprofit, and technology sectors in a virtual workshop on 

the use of remote research methods to study learning at home. 


+ Provide examples of how researchers are repurposing and 
reinventing qualitative methods for remote contexts with 
the aim of expanding and improving these methods for 
future studies; 

+ Highlight how families and communities are innovating and 
adjusting to the pandemic and how these adjustments are 
shaping learning and wellbeing in unexpected ways; 

+ Convey on-the-ground perspectives from caregivers and 

In this report, we summarize the strategies and insights 

generated at our July 2020 workshop so that we may share them 

among a wider network of researchers, practitioners, funders, 


learners about the challenges of remote learning and inspire 
solutions to solve them; and 
+ Mobilize collaborative efforts for future research and design. 


TABLE 1: NSF RAPID projects featured in this report 


Stanford University 


University of Washington 


University of Michigan 


NSF Award No. 


2028082 


2027525 


2028370 


Project Title 


Using remote diary methods to 
understand how families navigate 
emergency-driven homeschooling 
driven by COVID-19 


Understanding family life and the 
role of technology in the context 
of COVID-19 


How people learn rapidly: COVID-19 as 
a crisis of socioscientific understanding 
and educational justice and equity 


Principal Investigator(s) 


Brigid Barron 


Julie Kientz, Alexis Hiniker, 
Sean Munson, Jason Yip 


Angela Calabrese Barton, 
Elizabeth Davis, Leslie Rupert Herrenkohl 


NSF Directorate/ Division/Program 
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Social, Behavioral & Economic Sciences 
(SBE) / Behavioral and Cognitive 
Sciences (BCS) / Science of Learning & 
Augmented Intelligence (SL) 


Computer and Information Science and 
Engineering (CISE) / Information & 
Intelligent Systems (IIS) / Cyber-Human 
Systems (CHS) 


Education & Human Resources (EHR) / 
Division of Research on Learning (DRL) 
/ Advancing Information STEM 
Learning (AISL) 


About the Remote Methods Workshop 

On July 17, 2020, we hosted a 2-hour-long Zoom-based workshop 
featuring three NSF RAPID projects, all of which were studying 
how the Spring 2020 school closures were affecting family 
learning and dynamics. Notably, these projects were funded 
by three different NSF directorates—Social, Behavioral and 
Economic Sciences (SBE), Computer and Information Science 
and Engineering (CISE), and Education and Human Resources 
(EHR)—which speaks to the diversity of theoretical and 
methodological approaches used in conducting this work 

(see Table 1). In addition to the principal investigators (PIs) 

of the RAPID projects, we also invited a small number of 
researchers whom we knew were also conducting, or about 

to conduct, learning-related research using remote methods. 
The full list of workshop participants can be found in Box 1. 


To make the most of our brief time together, the three RAPID 
teams wrote and distributed 2-page summaries of their studies a 
few days before the workshop to give all participants a general 
sense of each study's aims, research questions, and methods. We 
also invited non-presenting researchers to serve as discussants 
and requested that they come prepared to comment on the focal 
studies. We spent the first half of the workshop discussing the 
focal projects as a whole group; the latter half took place in Zoom 
breakout rooms, where smaller groups could delve into the 
methodological challenges and opportunities of conducting 
remote research with families. At the end of the workshop, 
participants reconvened in a full-group brainstorm of how we 
might create a more formal alliance around our shared aims and 
activities as well as advancing scholarship and practice around 
the study of remote learning. 
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This report is similarly organized. First, we present the NSF RAPID 
projects as case studies of three varied approaches to documenting 
how learning is transpiring in American homes and communities, 
without compromising the health and safety of either subject or 
researcher. We next present an analysis of the methods employed 
by the cases, highlighting their common and unique challenges 
and opportunities. The report concludes with a set of insights 
gleaned from across the three studies, which may serve as a guide 
for researchers interested in conducting similar work. We also 
believe that educators, policymakers, and educational technology 
developers will find these insights useful to their decision-making 


as it pertains to improving learning in a post-pandemic world. 


BOX 1: Workshop participants 


+ Maria Alvarez, Common Sense Media 
+ Sheena Erete, DePaul University 
+ Matthew Kam, Google 
+ Catherine Jhee, Joan Ganz 
Cooney Center 
+ Kiley Sobel, Joan Ganz Cooney Center 
+ Lori Takeuchi, Joan Ganz Cooney 
Center? 
+ Michael Preston, Joan Ganz 
Cooney Center 
+Vikki Katz, Rutgers University 
+ Brigid Barron, Stanford University 
+ Caitlin K. Martin, Stanford University 
+ Cindy Lam, Stanford University 
+ Judy Nguyen, Stanford University 
+ Rebecca Silverman, Stanford University 
+ Rose Pozos, Stanford University 


+ Veronica Lin, Stanford University 

+ Craig Watkins, The University of Texas 
at Austin 

+ Ricarose Roque, University of Colorado 
at Boulder 

+ Angela Calabrese Barton, University 
of Michigan 

+ Betsy Davis, University of Michigan 

+ Day Greenberg, University of Michigan 

+ Leslie Rupert Herrenkohl, University 
of Michigan 

+ Tammy Tasker, University of Michigan 

+ Jason Yip, University of Washington 

+ Julie Kientz, University of Washington 

+ Rebecca Michelson, University of 
Washington 

+ Sean Munson, University of Washington 


? At the time of the workshop, Lori Takeuchi was on staff at the Joan Ganz Cooney Center; 
Takeuchi is now a Program Director at the National Science Foundation in the Directorate of 


Education and Human Resources. 
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STUDY 1 

Using remote diary methods to understand how families 
navigate emergency homeschooling driven by COVID-I9 - 
Stanford University 


NSF Award #2028082, Division of Behavioral and Cognitive Sciences, 
Directorate for Social, Behavioral and Economic Sciences 


By Brigid Barron, Caitlin K. Martin, Rose K. Pozos, Cindy K. Lam, 
Judy Nguyen, Zohar Levi, Susie Garcia, and Veronica Joyce Lin 


Background 

The speed of transitioning to remote instruction varied 
enormously from place to place, as did the quality, quantity, and 
form of home-school connections, raising significant concerns 
about equity. Parents and other adult caregivers at home were 
asked to take on new roles as co-teachers and facilitators of 
remote learning. According to an analysis of the Household 
Pulse Survey carried out by the U.S. Census Bureau, parents 
spent an average of 13 hours a week supporting their children’s 
learning during the spring of 2020 (Wang, 2020). Understanding 
the ways in which families adapted to the challenging conditions 
of the COVID-19 pandemic is critical to designing better solutions 
for the future: How did families support their child's learning, 
and what challenges did they face? What roles did technology 
play, and what equity issues emerged? What positive outcomes 
did parents observe? Our analysis foregrounds asset-based 
perspectives, attending to the resourceful ways that families 
adapted simultaneously to changes in normal routines and 
variation in access to resources (Lee, 2010). 


The study 

Our research team conducted a diary study that captured a 
diverse set of 109 families’ experiences during the first wave of 
U.S. school closures in the spring of 2020. Leveraging the design 
of a pre-pandemic pilot study of the diary-study method for 
examining home learning (Barron et al., 2021), we developed a 
remote-research approach, allowing parents, over the course of 
one week, to voice the impact that the COVID-19 crisis while 
also documenting learning opportunities in their homes over 
the course of one week. 


Methods 

Data collection was enabled by dscout, a smartphone-based 
qualitative research platform. With the aid of dscout, our research 
team could interact with families in near real time and collect rich 
qualitative data without face-to-face contact, affording broader 
geographic reach and pandemic-safe practices. The approach 
also has affordances for participants, including mobility and 
ease of uploading videos and pictures in real time. For example, 
a busy parent can snap a quick picture of their child doing an 
activity and upload it while also making dinner or during a break 
between meetings. Videos taken at home—with background 
noise from children and pets—provide rich context previously 
reserved for expensive and potentially intrusive home visits. 
Participants submitted a total of 1,103 unique entries over the 
course of the study, responding to researcher questions about 
five primary topics (see Appendix: Table 6), including 668 “diary 
entries,” which describe learning moments that happened 
during the week. Each participant submitted a minimum of six 
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diary entries. Each entry included multiple-choice survey items, 
open-ended text responses, image uploads, and video prompts, 
as seen in Table 2, which summarizes six diary entries over the 
course of a week from one of our participants. 


Findings 

Our team used a variety of analytical methods to explore the 
data, including: construction of indices reflecting breadth of 
challenges, diversity of resources used, and perceived learning 
outcomes; coding open-ended responses and transcripts to 
capture variation in themes using grounded theory and deductive 
approaches (see Appendix: Table 7); developing case portraits 

to help theorize parent-identified examples of learning; and 
comparative analyses based on demographics and other variables 
of interest. Here are some of our findings: 


+ Challenges. Parents described a diverse range of challenges 
associated with their new roles as co-facilitators of learning. 
Many struggled to coordinate their own work while also 
overseeing their child's academic tasks. Caregivers also grappled 
with pedagogical issues, realizing that they did not know how 
to teach particular content even when they found the time. 
Another cluster of concerns emerged around keeping their 
child engaged and motivated. Many children had a hard time 
focusing on schoolwork and keeping up with assignments. 
Finally, more than half of the caregivers described social and 
emotional concerns (Nguyen et al., 2021). Children missed 
their teachers and friends, and they worried about meeting 
academic requirements. 


TABLE 2: Data across diary entry submissions from one participant 


Monday Tuesday Wednesday Friday Sunday Monday 
Time Early afternoon Early evening Early evening Early evening Late afternoon Late morning 
(12-3PM) (5-8PM) (5-8PM) (5-8PM) (3-5PM) (9-12AM) 
Origins School School Parent Child Parent Parent 
Topic Writing Writing Math Science Other: Sounds Other: Light & heat 
Photo 
Ps 7 } , — 
fiat is 
Researcher Emi writes out words on Emi traces words in the Frustrated with the Emi wonders why she Emi identifies photos Emi identifies photos of 
summary of a worksheet. Mom helps worksheet then matches — school worksheet can’t hold water. Mom of loud and soft sounds sources of light and heat 
parentactivity = and has Emi “read it the word with a picture. packet, Mom asks sets up inquiry activity in the packet. Mom in a worksheet. Mom 
description over and over again Mom helps Emi sound Emi to arrange sticks to compare water extends this worksheet plans an extension to 
until she’s got it.” out words. into pairs to practice in liquid and solid activity by banging find things around the 
counting by 2s. forms, supporting things around the house house that light up. 
sense-making and and documenting their 
introducing new variation. 
vocabulary. 
Parent rating 3 5 5 10 10 10 
of learning 
(0-to-10) 
Parent rating 2 2 10 10 10 10 
of fun 
(0-to-10) 
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+ Parents as learning partners. Building on learning partner 
practices identified in earlier research (Barron et al., 2009), 
we asked parents how they were supporting learning. Sixty 
percent collaborated with their child, including participating 
in school assignments and facilitating innovative hands-on 
projects, and this was similar across income groups. Diary 
entries surfaced new learning partner roles, such as listening in 
on synchronous classroom instruction, troubleshooting tech- 
nology and managing workflow. (B. Barron et al., 2021). Entries 
also illustrated how caregivers arranged activities supporting 
engagement and extending what schools could offer. 


+ 


Technology and equity. At the time of the study, most school 
instruction, assignments, and communication with families had 
moved online across the country. Technology was critical for 
accessing materials and maintaining social connections. Even 
within our relatively well-connected sample, children from 
higher-income families were more likely than those from lower- 
income families to have synchronous virtual classes (95% vs. 
70%), access to video lessons (73% vs. 43%), and personalized 
communication with teachers (92% vs. 65%) (Pozos et al., 2021). 


+ 


Positive outcomes. Parents reported benefits (see Figure 1) 
ranging from more family time (94%) to closer teacher/parent 
relationships (41%). Many gained insights about their child’s 
learning (78%). Diary entries documented examples of how 
observing a child's classroom life sparked new ideas about how 
to support them. Some caregivers spoke specifically about 
using new knowledge to provide additional resources tailored 
to their child's needs. Others reconceptualized preexisting 
ideas about their child as a learner. 
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FIGURE 1: Benefits reported during the time when 


Yel sYoYo) Mn -1e- Mal (ox-Ke| 


100% 94% 
78% 771% 
75% 
50% 
41% 
25% 20% 
0% a 
Family time |More aware Child More New learning 


ofwhatand independence connected _ relationships 
how the child and new with school/ 
is learning life-skills teacher 


Implications for research 

+ Inspire designs for family engagement based on what worked 
well and what did not during the early months of the pandemic. 
Caregiver observations documented in the diary entries showed 
that they were attending to their child's feelings, interests, 
and understanding of content, providing crucial formative 
assessment data that most teachers are currently lacking and 
missing terribly. In future design work, tools and practices can 
be developed that allow caregivers and teachers to share insights 
about maintaining academic resiliency by building on children's 
interests and activity preferences, thereby sustaining engagement. 
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+ Develop models of remote research-practice partnerships 
(RPPs) with families, schools, and communities. Our study 
centered families as the core unit of analysis. Future research 
should involve multiple interdependent stakeholders, paving 
the way for continual improvement of remote and hybrid 
learning. Beyond schools and families, this work should include 
community-facing organizations (e.g., libraries, afterschool 


clubs, health agencies) that support learning (Erete et al., 2020). 


To contribute to the design of resilient communities facing 
simultaneous economic, educational, racial, and health-related 
inequities (Greenberg et al., 2020), it will be essential to design 
novel approaches that leverage existing learning networks 
and can honor RPP principles, including mutuality, trust, 

and co-construction (Coburn & Penuel, 2016). 
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Brigid Barron, PhD is a professor of 
education in the Learning Sciences and 
Technology Design program at Stanford 
University, where she studies how digital 
technologies can serve as catalysts for 
collaborative learning across home, school, 
and community settings. 


Caitlin K. Martin is a senior researcher with 
Barron’s lab at Stanford University and is 
an independent research and evaluation 
consultant focusing on out-of- school 
learning and data visualization. 


Rose K. Pozos is a PhD candidate at 
Stanford University in the Learning Sciences 
and Technology Design program, where 
she studies everyday family learning. 


Cindy K. Lam is a PhD candidate at the 
Stanford Graduate School of Education, 
with studies specialized in Learning 
Sciences and Technology Design as well as 
Developmental and Psychological Sciences. 


Judy Nguyen is a graduate student in 

the Learning Sciences and Technology 
Design program at Stanford University, 
studying how equitable formal and 
informal learning environments support 
students’ academic and social-emotional 
development. 


Zohar Levy is an undergraduate at 
Stanford University studying Linguistics 
with a focus on modern language 
(Arabic and Spanish). 


Susie Garcia is an undergraduate at 
Stanford University studying Economics 
and Philosophy with a focus on economic 
development. 


Veronica Joyce Lin is a doctoral student 

in the Learning Sciences & Technology 
Design program at the Stanford Graduate 
School of Education. Her research 
interests lie at the intersection of learning, 
young children, and technological and 
educational equity. 
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STUDY 2 
Education, work, and life during COVID-19: Supporting 
families at home with technology —- University of Washington 


NSF Award #2027525, Division of Information and Intelligent Systems, 
Directorate for Computer and Information Science and Engineering 


By Rebecca Michelson, Akeiylah DeWitt, Julie Kientz, Sean Munson, 
Jason Yip, and Alexis Hiniker 


Background 

As COVID-19 spread, American families urgently adapted to new 
ways of working, managing child and elder care, and facilitating 
remote learning experiences. This research aimed to understand 
parents’ experiences with technology in the home in the wake 
of millions of families suddenly taking ownership of their 
children’s education, often while working full-time or enduring 
the hardship of losing employment and seeking alternatives. 
Technology provided critical infrastructure, and new contexts 
required everything outside the family unit to now be done 
remotely while social distancing. We collected data on concerns 
affecting family life in these extraordinary circumstances 
through small, weekly activities in a shared online community. 
Our goals were to identify and share the greatest stressors 
families faced during these times, and design recommendations 
for improving everyday family tools such as remote learning 
technologies. Our research questions for this project included: 


+ Question 1: How are the roles of parents evolving during the 
pandemic? How are families meeting the competing needs 
of working remotely (if this is an option for them) and caring 
for children? 
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+ Question 2. How are families leveraging and adapting 
technology during this time, and what successes and 
challenges have they experienced? 


+ Question 3. What technology-supported potential solutions 
do families envision addressing their needs during times of 
crisis and prolonged social isolation? 


The study 

Our study applies the Asynchronous Remote Communities 
(ARC) method, which supports longitudinal engagement with 
harder-to-reach populations, communities who face stigma, and 
geographically disparate communities (Bhattacharya et al., 2019; 
MacLeod et al., 2016). Over 10 weeks, between May and August 
2020, we used Slack—an online chat platform used in a variety 
of business and community organizing settings—to ask families 
about their home technology needs, to refine their problem 
statements and identify resources, and to codesign solutions 
(see Figure 2). With the asynchronous nature of the study, 
families were able to connect with each other in three distance 
groupings across time zones, on their own schedules within 
moderated channels. While the study was planned to unfold 
over 10 weeks, we paused study activities for one week during 
the Black Lives Matter uprisings following George Floyd's 
murder to take time for reflection and learning. 
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FIGURE 2: Overarching themes of the Asynchronous Methods 
Remote Community method in this study, consisting After conducting nationwide outreach through our extended 
of discussion prompts, a diary study, interactive drawing networks as well as Facebook ads, we enrolled 30 economically 
activities, and codesigning sessions and racially diverse families with children ages 3-13 (preschool 
through 8th grade) and divided them into three groups from a 
screener with 320 responses (see Appendix: Table 8). The weekly 
prompts ranged from a diary study to community resource and 
information mapping to codesign activities, as illustrated in 
Table 3 on page 15. The Slack group data includes conversation 
Weeks 1-4 Weeks 5:7 threads, artifacts (i.e., love and break-up letters addressed to 
Understanding Refining technology), and drawings from the codesign activities. During 
oe ae deus the codesign weeks, we included the Mixing Ideas method, 
which structures collaboration by encouraging participants to 
combine their ideas with other participants. Mixing Ideas can be 
effective in settings where participants may have less confidence 
or social cohesion; this is because it requires participants to 
Weeks 8-10 review and build off of each other’s work. In fact, many families 
rae ce positively commented on each other's creative contributions 
and created new ones. Our team also conducted compensated 
follow-up interviews and coded the data through an iteratively 
developed codebook and grounded theory methods (Strauss & 
Corbin, 1997). 


ARC activities 


naming assets 
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TABLE 3: Activities for the 10 weeks of the Asynchronous Remote Communities 


Phase Week Activity name Prompt details Generative or recall 
Understanding il Introductions and advice After introducing themselves, parents were asked to Recall 
participant needs share what advice they would have given themselves 
pre-COVID-19. 
Understanding D Diary study Participants completed five different diary entries on Recall 
participant needs their technology use. 
Refining the problems 3 Ranking and ranting and writing a Participants reviewed a list of top technology-related Recall and Generative 
and benefits of letter to technology concerns and benefits (generated from screener survey 
technology use responses and the diary study entries). They ranked the 
concerns and wrote a love or break-up letter to a piece of 
a technology. 
Refining the problems 4 Information and resource mapping Participants created diagrams of their information flows Recall 
and benefits of and resources related to work needs, remote schooling 
technology use (or summer/after-school activities), and COVID-19. 
Study pause 5 Study pause Shortly after the murder of George Floyd, we held a 
study pause for our participants and research team to Generative 
reflect and protest accordingly. 
Codesign 6 Ideation Through partnered brainstorming, participants created 
solutions to address some of the most chaotic moments Generative 
of the pandemic. 
Codesign it Idea refinement Participants selected their top ideas and refined them 
with product names, descriptions, and sketches. Generative 
Codesign 8 Mixing ideas: Round 1 Participants created family technologies about 
COVID-19, supporting quality family time, addressing Generative 
anti-racism, or anything else that felt meaningful to 
them, based on combining ideas shared by others. 
Codesign 9 Mixing ideas: Round 2 Participants completed the “mixing ideas” prompt one Generative 
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more time with the latest batch of ideas shared. 


eS) 


Findings 


+ 


+ 


Improving remote learning systems. Families shared many 
requests to improve remote learning systems and other 
technologies and to adjust learning expectations. These include 
refining features that support finding, tracking, and updating 
school assignments as well as clearly communicating critical 
information (such as login information, classroom links, and 
expectations for academic progress). Parents also requested 
remote learning with more child autonomy, easing the burdens 
of parental supervision. We anticipate that these needs will 
persist, even once children return to school. 


Equity considerations. Based on learning about lack of support, 
we found that children who need extra attention, such as 
those with learning disabilities, faced the greatest risks of 
falling behind. Parents of these children reported a greater 
need to provide consistent supervision during coursework and 
homework, often at the cost of skipping work or caregiving for 
others. Another barrier, especially for families in our lower SES 
group, was limited access to devices at home. One parent told 
us that her two sons had to share her cellphone to attend class 
because they did not have any other Wi-Fi-enabled devices. 
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Implications for research and practice 


+ 


+ 


+ 


We chose the moderated Asynchronous Remote Communities 
(such as through Facebook groups, Discord, Slack, etc.) because 
they enabled quick deployment in a rapidly evolving and 
uncertain context while maintaining deliberateness with our 
prompts (MacLeod et al., 2016). For example, we began the study 
with a concrete plan of initial activities and adapted the prompts 
as needed based on our pandemic learnings. ARCs also offered 
space for personal story sharing, learning from others in similar 
circumstances, and creative explorations. Our team selected 
Slack over other social media tools because participants could 
choose to participate anonymously with pseudonyms. 


The majority of parents we interviewed shared that they 
enjoyed the experience of taking part in the study and 

would participate in a study like this again because it offered 
opportunities for creative brainstorming and fun experiences 
with reviewing others’ ideas. Future ARC themes could also 
include different stakeholders, such as teachers, parents, and/ 
or learning technology designers, with the aim of building 
empathy and understanding through facilitated activities 
and the development of prototypes. 


While improving technology design might alleviate some of the 
learning issues families experienced, many of the technological 
problems encountered were influenced by strategic decision- 
making, such as standardization of technology use for teachers 
within a school or district. With the implementation, if 
possible, of IT support or a technology troubleshooting hotline, 
schools may gain a better understanding of not only how but 
also why families experience technology issues. 
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+ Technology loan programs can address device gaps between 
less- and more-privileged families, and social worker support can 


assist with connections to social services (such as navigating 
SNAP/EBT programs, unemployment benefits, etc.). Special 
efforts need to be made to accommodate students with 
Individualized Education Programs and limited device and 
broadband access for remote schooling. 
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Background 

In long-term research-practice partnerships (RPPs) in the 

Midwestern and Western United States, we sought to understand 

what/how community partners learn about and take action 

on COVID-19 and justice-related concerns. Our overarching 

research questions include: 

+ How and what science do people learn about COVID-19? 

+ How do people activate and apply the science they learn to 
make (or revise) personal and family decisions? 

+ How is youth and adult learning about COVID-19 shaped by 
individuals’ critical consciousness around racial, educational, 
and economic justice? 


As we sought to answer these questions, we further considered 
the political and ethical dimensions of doing research through 
remote methods. 


We draw upon theories of consequential learning, focusing on 
what counts as valued learning, as well as how the processes of 
learning involve disrupting or transforming normative patterns 
of participation towards new forms of expertise and social 
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relations that counter ahistorical and universalist notions of 
knowing (Gutiérrez et al., 2019). Consequential learning reveals 
how learning matters to people both here and now and in 
imagined social futures. It also illuminates power dynamics 

in how actors are positioned across time, place, and scales of 
activity (Jurow & Shea, 2015). To attend to power in our remote 
methods, we further draw upon critical witnessing and being 
with—practices toward conscientization, social transformation, 


Methods 


and the public good of communities historically marginalized by 


systemic inequities. We seek to amplify possibilities for coalition 
building and learning in “daily, (extra)ordinary, and intentional” 
work (Villenas, 2019, p. 153). 


TABLE 4: Remote methods 


Approach 


Generation 


Focus 


This larger study involves 60 participants across two metropolitan 
areas representing different U.S. geographic regions (see Appendix: 
Table 9). We used both remote participatory interviewing and 
informal conversations with experience sampling methods, as 
outlined in Table 4 below. Methods took shape interactively as 
we partnered with participants to cogenerate data over time 
through a multimodal and dynamic data cogeneration design. 


Remote methods equity considerations 


Dialogic interviews 


4 interviews/participant, 
90-240 minutes each 


What COVID-19 information individuals access/ 
apply towards decision-making, how, and why; 
Personal/community COVID-19 experiences; 
Use of resources and social networks; Critical 
political awareness, clarity; action taking 


Informal conversations 


Organic spillover from interviews 
led by participants 


Share their complex and layered stories in 
multimodal ways, giving depth to each of 
the layers, while also capturing interactions 
among layers 


Range of tools: phone, video conferencing, text 
Range of contexts: One-on-one to whole family 
Co-strategizing interview times/days/structures (e.g., 
breaking up interviews over separate days if needed) 
Co-constructing timeline using multi-modalities 


Critically being with and witnessing over time 
via informal texting, social media link sharing, 
photo/video and meme sharing 


Experience sampling 
method 


Monthly Google surveys, sent to 
participants via text and/or email 
with brief reminders to share any 
relevant updates 
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What updates participants experience between 
interviews 


Asynchronous opportunity to bear witness and to 
continue centering participant voices over time 
Personal reflections, social media link sharing, 
photo/video and meme sharing, narrative-building 
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FIGURE 3: Experience Sampling: Examples of TikTok 
Nite C=oS-My o} (eXe Le (-Ye Mo’ eel ai ral] Lela ey 


F 


Laughing at corona virus | 
memes when it reached 
Italy | 


Our long-term relationships framed and mediated our data 
cogeneration. We built relationships of trust and shared 
vulnerability in our research-practice partnerships over years 
or decades, which laid a foundation for us to develop the multi- 
modal and organic elements of our design. We co-analyzed data 
with participants using critical inquiry/grounded theory, in a 
constant comparative, continuities/contradictions approach 
(Charmaz, 2017). Initial coding categories for our grounded work 
are outlined in Table 10 of the Appendix. 


Our approach is shaped by our efforts to unlearn, relearn, and 


remix research tools we have used in the past, towards opening 
up possibilities for being with/critical witnessing as a part of 
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this research process turned remote. For example, we met with 
long-term partners via text and phone calls to co-strategize 
what methods youth/families are comfortable using and/or 
learning, as well as when, how, and for what purposes (e.g., 
co-planning for a video-based dialogue during family dinner). 
We also used remote methods to meet research and life needs, 
such as collaboratively sewing masks and coordinating supply 
sharing/distribution during video-based interviews. 


Findings 

We share our insights thus far in three separate papers on the 
topic of how people's learning is shaped by justice concerns and 
social context during a time of crisis. These papers address 
community infrastructuring (Greenberg et al., 2020), critical data 
practices (Calabrese Barton et al., submitted), and the role of 
trust in building knowledge and taking action (Herrenkohl et al., 
in preparation). 


First, participants came together to co-create new data- 
constructing and data-sharing infrastructures that could better 
serve their needs as they continued to learn the science of 
COVID-19 and its equity-related complexities (e.g., Greenberg et al., 
2020). By community infrastructuring, we mean justice-oriented 
acts of necessity for community protection and wellbeing; these 
acts involve sharing and reorganizing resources in the moment 
and with an eye towards the urgent near future. Participants 
highlighted conflicts between sources and messaging, where 
and how they saw their own experiences mirrored/challenged 
in such messaging, and the extent to which these data could 
possibly inform/transform their decisions and actions. 
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Using multimodal forms of information sharing with us, 
participants pointed out how misinformation campaigns 
fostered deepening racial and socioeconomic injustices and 
what that meant for their own practices. 


Second, participants’ enactment of critical data practices changed 
how they negotiated between coming-to-know and coming-to-act 
(Calabrese Barton et al., 2021). Community-engaged critical 
practices are what people do in socially-mediated and culturally- 
embedded ways with, in relation to, and oriented around data 
and data infrastructures in support of more equitable everyday 
living and communities (Milan, 2019). These practices involved 
youths ' efforts to recognize and leverage their intellectual power 
to participate in and challenge real and consequential aspects 

of everyday living and learning in a pandemic as STEM-agentic 
people. These practices arose from tensions in their engagement 
with data as they mobilized big and small data from different 
epistemological and social origins towards meaning-making, 
action-taking, and communicating. These tensions shaped how and 
why youth critically navigated, leveraged, and critiqued big data 
to create meaning. For example, youth remixed, recontextualized, 
and repositioned big data through the lenses of small data as they 
sought to bring provenance, utility, and visibility to their meaning- 
making. They engaged these practices towards liberatory effects 
in how they navigated and reimagined data regarding their 
hoped-for worlds. Youth not only reimagined data as sites of 
struggle over what and who counts in the developing data-rich 
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narrative of COVID-19 and its intersections with justice-related 
concerns, but they also enacted alternative infrastructures for 
counter-data production and aggregation towards justice in 
both the here-and-now and possible futures. 


Third, participants’ coming-to-know and coming-to-act were 
mediated by issues of trust in relation to what one knows and 
seeks to know, as well as who one is in relation to the community 
(Herrenkohl et al.,in preparation). Cognitive models of learning 
suggest that constructing strong foundational knowledge about 
COVID-19 leads to adopting practices, such as mask wearing, for 
preventing viral spread. This approach fails to recognize the role 
of culture and context in shaping these practices. For instance, 
early in the pandemic, we found that Asian- or Asian American- 
identifying participants who brought prior knowledge/experience 
with other pandemics and the practice of mask wearing quickly 
understood that the practice of mask wearing is helpful for 
preventing viral spread in the community. Yet, some participants 
reported not wearing masks early in the pandemic because 
mask wearing was then interpreted as an indication that 

“Tam sick” rather than “I am being responsible community 
member.” We found that although participants built scientific 
understanding from trusted sources and experiences, they did 
not trust the general public, fearing racism, intimidation, and 
violence. Findings suggest that we need new models of learning 
and action that recognize cultural contexts, practices, and 
systemic injustices. 
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Implications 

Across findings, youths’ learning about and decision-making 
related to COVID-19 is deeply tied to how, when, and why they 
access data and data infrastructures in relation to their lives and 
communities, as well as how trust mediates these processes. 
Attention to political and ethical dimensions of learning, and to 
how tools get leveraged, remixed, and co-opted in the crucial 
work of critical witnessing and being with, are requirements for 
advancing justice-oriented work with community partners in 
remote research. 
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Guide to remote 
aal=sialeless 


In this section, we offer direct comparisons of the methods that 
researchers at Stanford, the University of Washington, and the 
University of Michigan innovated in the face of COVID-19 to 
maintain human connections with their study participants when 
face-to-face interactions became no longer feasible. In doing so, 
we hope to (a) highlight the strengths, weaknesses, opportunities, 
and challenges presented by this select set of approaches, and 
(b) inspire thinking around how we might add to and improve 
upon them. By improving the methods for studying learning at 
home and in communities, we believe we can more faithfully 
document the experiences of traditionally underserved families 
and more effectively advocate on their behalf to education 
leaders, policymakers, funders, and media and technology 
developers. While other researchers interested in studying 
learning in restricted environments will find this guide useful, 

we consider its contents a mere starting point for what we hope 
will be an ever-growing repository of methodological wisdom 
and sharing across disciplines and sectors. 


The table on page 23 provides side-by-side comparisons of 
the methods employed by the three studies. The section that 
follows synthesizes commonalities across them, illustrating 
the opportunities and challenges associated with conducting 
remote research using online tools. 
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TABLE 5: Comparison of remote methods (continues on page 24) 


Stanford 


University of Washington 


University of Michigan 


A. Primary goal/ 
question 


+ How did families support their child’s learning 
and what challenges did they face? 

+ What roles did technology play? 

+ What positive outcomes did parents observe? 


+ How did the role of parents evolve? 

+ How are families leveraging and adapting 
technology? 

+ What technology-supported potential 
solutions do families envision as addressing 
their needs during times of crisis and 
prolonged isolation? 


+ How and what science do people learn about 
COVID-19? 

+ How do people activate and apply the science 
they learn to make (or revise) personal 
family decisions? 

+ How is learning about COVID-19 shaped 
by critical consciousness around racial, 
educational, and economic justice? 


B. Time frame 


2 weeks (May-June 2020) 


10 weeks (May-August 2020) 


1 year (March 2020-April 2021) 


C. Recruitment 


Inviting existing dscout participant pool to 
complete screener 


Inviting extended network and recruiting through 
Facebook to complete a screener 


Leveraging deep and long-term relationships 
with participants prior to study 


D. Participants 


109 parents/caregivers talking about and 
sometimes including their 5-10-year-old child(ren) 


30 families with 3-10-year-old children 


60 participants, including adults, young adults 
(ages 19-24), and children (ages 0-18) 


E. Methods 
overview 


Asynchronous 

+ Diary studies 

+ Survey measures 

+ Reflective prompts inviting video and 
image responses 

+ Ratings of learning moments 


Asynchronous 

Asynchronous Remote Communities (ARC) 
incorporating: 

+ Conversation prompts 

+ Diary studies 

+ Interactive drawing activities 

+ Codesign sessions using mixing ideas method 


Synchronous 
+ Dialogic interviews 
+ Informal conversations 


Asynchronous 
+ Experience sampling 


Synchronous 
+ Interviews 
F. Remote data + dscout (mobile app) + Slack + Telephone 
collection tools + Mobile phones + Zoom + Text 
+ Google Forms 
+ Zoom 
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TABLE 5: Comparison of remote methods (continued from page 23) 


Stanford 


University of Washington 


University of Michigan 


G. Participant- 
to-researcher 
interactions 


Researchers commented on participant diary 
entries and other submissions, resulting in 
threaded discussions 


Researchers engaged in discussions and 
interactive activities with participants on Slack 


Researchers co-analyzed data with participants; 
co-strategized methods to use with youth/ 
families; engaged in non-research activities such 
as sewing masks and coordinating supplies 


H. Data records 


+ Video of participant responses (self-recorded) 
+ Shared artifacts (images) 

+ Survey responses 

+ Threaded discussion transcript 

+ Transcripts of video-entries 


+ Video of participant responses (interview) 
+ Shared artifacts (drawings, documentation) 
+ Threaded discussion transcript 

+ Drawings from codesign activities 


+ Video and audio of participant responses 
(interview) 

+ Shared artifacts (links, memes, photos/videos) 

+ Survey responses 


I. Analysis + Comparative analyses of quantitative data + Open coding using grounded theory + Collaborative coding with participants using 
+ Summative metrics + Iteratively developed codebook critical inquiry/grounded theory 
+ Visual and narrative case portraits 
+ Open coding using grounded theory 
+ Iteratively developed codebook 
J. Approach + Geographic diversity of participants + Smaller sample to capture depth of experiences + Geographical focus on single communities 
highlights + Larger sample to capture breadth of experiences + Collaborative design + Participant-researcher collaboration 


+ Mobility of data collection device 
+ Rich data (e.g., video) offers insights into 
context of homes 


+ Participant interaction 


+ Equity considerations with remote methods 
+ Made possible by multiyear research practice 
partnership 


The opportunities and challenges of remote research methods 
When schools shuttered in the Spring of 2020, researchers at 
Stanford, the University of Washington, the University of 
Michigan, and certainly dozens of other institutions leapt at 
the opportunity to investigate how the sudden lockdown would 
alter formal and informal learning as we had come to know it 
here in the United States. Many were as eager to figure out how 
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to study the evolution of learning at home at a time when it was 
no longer safe to spend time observing or interviewing learners 
in person. Others were forced to adjust their data collection 
protocols to gather data remotely and/or cultivate the trust of 
strangers whom they would likely never meet face-to-face. These 
adjustments, at least for the three teams featured in this report, 
involved quite a bit of ingenuity when it came to adapting 
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existing research and communication technologies (see Row F of 
Table 5 above) to get into the field swiftly enough to document 
family learning as it unfolded within the first few months of the 
pandemic. Yet, despite the convenience and flexibility of online 
tools in providing researchers with safe and relatively intimate 
access to participants, those tools also posed challenges—both 
novel and familiar—to researchers and participants alike. 


Based on conversations from the July 2020 workshop and our 
analyses of the three case studies, here we present a set of the 
most salient opportunities and challenges specific to employing 
online tools for conducting remote qualitative research. 


Opportunities 

Access to families 

+ Greater reach. New and greater forms of access to families and 
their lives was a significant opportunity presented across the 
studies. In all three, researchers recruited a range of participants, 
hearing and making sense of critical variations in how people 
learn, taking into account the ways that learning relates to 
access, opportunity, preparation, social networks, and ideologies. 
Remote data collection tools afforded researchers the ability to 
recruit and include participants across geographies to a greater 
degree than would be possible using traditional methods 
constrained by researcher-participant proximity. Within two 
weeks, the Stanford team used dscout to recruit and engage 
a geographically-distributed pool of more than 100 parents 
and other caregivers representing a range of household 
income levels. The University of Washington team designed 
and deployed Facebook ads to recruit and organize over 30 


GUIDE TO REMOTE METHODS 


families with variety related to race, household composition, 
home income, and connectivity. 


+ 


Convenience to families. All three research teams made use 

of asynchronous online platforms that allowed participants 
to contribute when and wherever was most convenient to 
them. They could upload diary entries, complete online surveys, 
or respond to email messages before the rush of readying the 
kids for online school, on the bus, or during downtime at work. 
Even texts and Slack, platforms that afford near-real-time 
conversation, made it possible for participants to handle 
researcher prompts and inquiries at will rather than as 
interruptions to their otherwise busy days. This gave participants 
the time and space for deeper reflection on their assignments 
than if administered in face-to-face interviews, which researchers 
have always had difficulty scheduling. While asynchronous 
methods are no replacement for real-time conversations, their 
convenience may also lessen the likelihood of participant attrition. 


Intimacy and connection 

Through the use of remote methods, the type of access researchers 
had to participants also expanded. Researchers reported rich 
phenomena, using pandemic-safe practices without sacrificing 
detailed situational data including sights and sounds of the 
home. While this intimacy also raises ethical challenges to be 
described below, the data reflecting people's authentic lives-in- 
the-moment is powerful and contributes to better understanding 
what families are facing during this time where day-to-day 
practices have shifted significantly. Despite concerns that the 
human element of qualitative research might be diminished 
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when interactions are completely mediated—via Slack, FaceTime, 
or even phone calls—all three teams reported high levels of rapport 
with participants. The near-real-time quality of texting and 
Slack platforms allowed for more frequent, conversational, and 
casual correspondence between parties, which may have helped 
establish a level of comfort and trust that periodic in-person 
visits and even email exchanges common in pre-pandemic 
studies may not have otherwise afforded. 


These communication tools enabled the University of Michigan 
team, for instance, to engage in ongoing dialogue with key 
informants, thus allowing them to both chart the direction 

of their study based on community needs and collaborate on 
activities as fellow community members—such as sewing masks 
and distributing supplies—resulting in new relationalities 
between researchers and participants, such as invitations into 
homes, new/deeper personal connections, and humanizing 
conversations. The University of Washington participants 
expressed appreciation for the reflective and creative aspects of 
the study, also made possible by the asynchronous nature of the 
online tools, which gave them the time and space to tackle their 
assignments without the pressure of researchers peering over 
their shoulders or keeping time, as real-time visits might impose. 
In both cases, participants may have felt less like subjects to be 
studied and more like colleagues with valuable perspectives. 


An abundance of data 

+ Multimedia formats. The ability to collect a large multimedia 
dataset, including rich qualitative data, in a relatively short 
amount of time was also considered highly valuable and unique 
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to remote methods. Researchers embraced the multimodal 
possibilities of remote data collection, accumulating, in all three 
studies, various data artifacts representing what participants 
were doing and thinking, including video, images, social media 
posts, chat dialogue, memes, and links to news stories. 


Automated organization. The automated digitization and 
transcription features embedded in remote tools minimized 
researcher hours and maximized data collection and organization. 
Although data cleaning and manipulation is still required, some 
of the heavy lifting is done within the remote systems. Chat 
and discussion threads are archived as participant-identified 
records of interaction, video is automatically transcribed, data 
across different entries are merged according to participant, 
and participant-uploaded artifacts are time-stamped and 
saved in digital folders. 


Living artifacts. Finally, remote methods afforded participant 
reflection and commentary on the data as it was collected, 
thereby creating another source of data. The University of 
Washington team encouraged participants to combine their own 
ideas with those shared by other participants, reviewing and 
building off work, including positive commenting and iterative 
design versions. The University of Michigan team created 
timeline representations during the study and asked participants 
to reflect and interpret, serving as an anchor for further data 
collection and interpretive refinement. Triangulation of these 
rich data were used to deepen understanding of learning 
moments and make connections between participant reflections 
and what was going on in the world. 
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Challenges 
Logistical dimensions 


+ 


+ 


+ 


Dataset size. The studies that utilized participant-led, 
asynchronous remote data collection quickly amassed a data 
corpus larger than what would be possible with traditional 
family ethnographies or design workshops. While bountiful 
data is also identified as an opportunity, finding new ways to 
organize and analyze it quickly and efficiently was something 
researchers grappled with regularly. 


Cost of tools. While several of the tools researchers used to 
collect data are free or low cost (e.g., Slack) or are part of 
university subscriptions that allow researcher access (e.g., 
Zoom), Stanford used dscout, a more sophisticated but more 
expensive tool that was specifically designed to conduct 
remote qualitative research. Cost was a consideration in 
thinking about not only what was possible for this particular 
phase of work, but also subsequent new or follow-up studies 
during the pandemic. 


Coordinating times for systematic data collection. While 
asynchronous methods allowed for researchers and participants 
to participate when convenient to them, other factors beyond 
researchers control affected scheduling. City, state, and national 
guidelines related to the pandemic changed from day-to-day, 
affecting how people lived and worked; additionally, a powerful 
period of civic unrest related to police violence emerged during 
the studies. Persisting with data collection schedules and 
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encouraging participation during a stressful time was sometimes 
tricky without in-person conversations about what was working 
and how things were going, especially in situations where 
researchers did not know participants beforehand. 


Ethical dimensions 
+ The intimacy associated with views of authentic living spaces. 


Although access to participant home environments is not 
unique to remote methods, data collected without the 
researcher being physically present blurred the boundary 
between structured research and something more personal. 
Some participants recorded their asynchronous video responses 
at the end of the day right from their bedrooms. For others, 
their kitchen dishes, dogs, and to-do lists were on full display 
during interviews. These intimate background portraits have 
the potential to reveal unanticipated views, including ones 
that participants are unaware of showing, raising questions 
about what can and should be recorded. Researchers mitigated 
this issue to some degree by having participants self-document 
or by adjusting data collection protocols with participants’ 
input to better shield their privacy. 


Questions over data ownership. A related issue is that of data 
ownership when researchers are using a third-party system 
for collection (e.g., Zoom, dscout, and Slack). While researchers 
could offer clarity to participants over how they would be 
using and sharing the collected data as sanctioned by their 
institutional review board (IRB) approvals, they could not make 
the same reassurances on behalf of the online tool providers. 
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+ Navigating new rules. Researchers also found themselves 
in the unexpected position of navigating new and complex 
organizational rules of conducting research during a pandemic 
as set forth by university IRBs as well as city and state 
guidelines. Sometimes, as with the University of Washington 
team, these rules conflicted with their humanistic sensibilities 
to do what was best for the community under study during 
times of crises, such as making masks together and participating 
alongside participants in marches for social justice. 


Access and inclusion considerations 

Workshop attendees voiced particular concern over access 

and inclusion-related issues as they pertained to families that 
couldn't take part in their studies because they lacked access to 
or weren't regular users of prerequisite technologies. For instance, 
the Stanford study's sample was drawn from the dscout database, 
which comprises 100,000+ “scouts” who have signed up to take 
part in smartphone-administered market research studies, for 
which they are offered monetary compensation (dscout, 2020). 
This assumes ownership of a smartphone, some level of 
tech-savvy, and a reliable Internet connection. As such, while 
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the Stanford study included some families that reported 
inconsistent or unstable Internet, none were entirely without any 
Internet connectivity. Similarly, the University of Washington 
researchers ARC method focused on participant interactions in 
online communities such as Facebook, Slack, and Discord; social 
media nonusers, therefore, did not participate. For the purposes 
of their small-scale qualitative study, representation of the full 
range of tech experience wasn't necessary; however, workshop 
attendees questioned the extent to which online tools may limit 
“under-connected” families (Rideout & Katz, 2016)—which tend 
to be overrepresented among lower-income, rural, and immigrant 
communities in the United States (Rideout & Katz, 2016; Vogels 
et al., 2020)—from participating in remote research due to 

lack of access and/or comfort with technology. Workshop 
attendees also voiced concern for undocumented families, who, 
understandably, may opt out of studies involving online tools 
for fear of being tracked or discovered by agencies such as the 
U.S. Immigration and Naturalization Service. Here again, the 
online nature of remote research may restrict participation of 
diverse families who have the most to lose by remaining unseen 
and unheard. 
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BLU Le liare mere) exelent hy 
for remote 
learning: Future 
(stere] ceamejare 
design priorities 


The July 2020 workshop focused on research questions, methods, 
and early findings from three studies that were conducted a 

few months into the COVID-19 pandemic. Drawing from the 
individual case study findings, our analyses of methods across 
the studies, and conversations from workshop, we present a set 
of research and design priorities for researchers, policymakers, 
technology and media producers, and funders to consider as we 
build capacity for remote learning and research moving into 

the post-COVID era. Priorities are organized according to the 
primary goals of building collective understanding across research 
findings and strengthening capacity for remote research. 


Build collective understanding of findings from COVID-I9 
research on learning and education 

In addition to the dozens of education-focused studies funded 
by NSF's RAPID COVID program, there are likely hundreds of 
other researchers in the United States and abroad investigating 
teaching and learning in quarantine. Some are using ethnographic 
methods, while others are administering surveys or carrying out 
controlled studies. All, however, are yielding knowledge that 
may benefit education practice, policy, technology design, and 
subsequent research, and this collective knowledge would surely 
be more impactful than the sum of its parts. Yet, as is too often 
the case in the scholarly world, this global community remains 
disconnected. Unfortunately, there is too much at stake for learners 
today to allow researchers to carry on in their silos, and so we 
propose the following suggestions for cultivating collaboration 
across methodological, disciplinary, and sectoral boundaries. 
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+ Locate, build, and strengthen networks across the global 
scientific community. While COVID-19 has in some ways 
improved scholarly sharing—webinars abound and in-person 
annual conferences have been replaced by online formats that 
make attendance more feasible from afar—a fundamental 
question still exists: How do we locate others doing similar 
research? Fortunately, there are resources like the COVID 
Information Commons (CIC), which provides a database of 
all COVID-related studies funded by the National Science 
Foundation (NSF #2028999), but searches are limited to 
work being conducted in the United States. More can be 


done to improve international sharing, communication, and 
collaboration around both our findings and methods of 
investigation. If the three cases studies featured in this report 
have taught us anything, it’s that online tools can, in fact, 
support meaningful and sustained relationships between 
researchers and participants. How might we use some of the 
lessons learned through our examination of remote methods 
to cultivate a more connected global scientific community? 


+ 


Synthesize findings across COVID-19 studies. More synthetic 
work is needed to integrate findings from studies that 

were independently conceived but that contribute to our 
understanding of what happened during the early phases of 
remote learning. These syntheses are particularly important 
now, as they may help practitioners improve the design of 
formal and informal learning systems and suggest avenues 
for future research. 
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+ Conceptualize equity-focused research and design agendas. 
Pandemic disruptions have led to a set of interdependent and 
multidimensional challenges to equitable access to learning 
opportunities. Research efforts toward the start of school 
shutdowns aimed to ensure that families’ immediate needs 
were addressed and to identify critical gaps in learning 
opportunities. Educational leaders have since noted that the 
dramatic shifts in learning routines, practices, and resources 
surfaced through this early-stage research have made it 
possible to reimagine educational systems (Darling-Hammond 
et al., 2020). To create a transformative research and design 
agenda, justice-oriented frameworks that address the political 
and social dimensions of racial and economic inequality are 
needed, along with associated conceptual work to define 
features of resilient educational systems and the conditions 
that support them. Research-practice partnerships that include 
families, communities, schools, and informal learning institutions 
will be essential for tailoring research and design efforts to 
address highly localized challenges facing communities. 


Strengthen the field’s capacity for rapid, collaborative, and 
useful remote research 

The Stanford, University of Michigan, and University of 
Washington teams drew on existing technologies and adapted 
their usual research practices to design the remote studies. 
During the workshop, attendees had several ideas for additional 
methods, tools, and norms that could better support participants 
and researchers alike, highlighting the need for shared norms 
and guidelines informed by collective stakeholders. 
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Innovate methods and prioritize funding for studies that 
promote inclusion. Preexisting research panels, social media 
networks, and other off-the-shelf solutions made it convenient 
and cost effective for PIs in this report to hit the ground 
running—especially within the constraints of the NSF's 
RAPID funding program—but these choices excluded certain 
families. It is critical to develop research approaches that 
include all families. Researchers need sufficient budgets to 
provide under-connected families with the tools needed to 
study them—namely, hardware (e.g., laptops), broadband (e.g., 
mobile hotspots), and requisite training. We also need to think 
out-of-the-box to design data collection tools and protocols that 
are, in fact, less reliant on technologies and practices not part 
of the everyday lives and realities of vulnerable families. These 
tools and protocols must also be developed with consideration 
for the needs and wishes of immigrant families regarding 
surveillance. Researchers must be absolutely transparent 
about who will have access to families’ data if they hope to 
assuage any fears or uncertainty that would prohibit certain 
families from participating. 


Develop research-practice partnerships with families at the center. 
Long-term, mutually beneficial collaborations between 
researchers and practitioners researching authentic problems 
are successful in producing actionable findings (Coburn & 
Penuel, 2016). But such partnerships rarely include families and 
parents as full collaborative partners. The need to recognize 
families as design and research partners in child learning is 
not new, but it is even more critical during school closures. 
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+ 


Workshop participants expressed particular interest in creating 
shared representations during the data collection process, 
such as event timelines or storylines that participants 

and researchers can see and build on together. Additionally, 
involving families, practitioners, and qualitative researchers 
with industry partners in discussions of what remote learning 
tools can and should look like has potential for informing rich, 
collaborative design and research work that integrates different 
interpretations of what is needed, what is valued, and what 

is possible. 


Establish new approaches and best practices for ethical and 

robust remote research methods. Technology-enabled qualitative 

research is, as we have discovered, complicated, and especially so 

when studying vulnerable populations. It requires the help and 

perspectives of other researchers, families, and social support 

organizations to identify the most pertinent pain points and 

weigh in on potential solutions. The July workshop on remote 

methods elicited such feedback, including suggestions to: 

- Train researchers on human-centered research methods 
(e.g., trauma-informed approaches) and consult participants 
on research and design decisions; 

— Establish new and/or expanded ethical norms around data 
collection confidentiality and privacy; and 

— Encourage purposeful collaboration between designers and 
researchers to design tools that meet both industry and 
academic standards in terms of cost, flexibility, data privacy, 
and ownership. 


31 


+ Reconcile the relationship between responsiveness and rigor. 
“Rigor” is acommitment to methodological procedure valued 
in both qualitative and quantitative research (Gill & R Gill, 
2020), and commonly pitted as the opposite of flexibility. But if 
the ultimate purpose of the scientific process is to improve the 
human condition, then researcher responsiveness to human 
needs—whether those needs arise from global, local, and/or 
personal circumstances—must be valued right alongside rigor. 
While qualitative research is more typically critiqued for 
sacrificing rigor for responsiveness, the pandemic has also 
pushed quantitative researchers to negotiate these tradeoffs. 
For instance, pharmaceutical researchers have departed from 
established methodological norms around COVID-19 vaccine 
clinical trials to get products to market sooner with the hope 
of saving more lives (Mukherjee, 2020). We argue that there is 
space for rigor and flexibility to coexist within high-quality 
research, and we encourage the broader scientific community 
to collectively redefine their coexistence in terms that no 
longer employ words like “sacrifice” and “tradeoff.” 
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In this report, we have shared three studies 
that foreground the experiences of families 
iiceLollivedave]|Nmelatel=)a-X-1a\c-10 Ml 0) OSE =eLUleeni(e) a 
systems: families whose children have 
suffered most from the pandemic-related 
Yel aToXo) mal (oXU] c=1ome ave mine] mane Melce (= 1a 
need our attention. Even in this advanced 
icon) ave) (ote) (ece] Mele (- Mm anlli ela smelmcy(Ule(s] al cm alen icc) 
essentially lost a year of school due to a 
Koel ato) mol oet=\-s-9m Comm CoXe) om ale] maTele](e Mo) Vm ALON WA 
be universal—a right, not a privilege. 


As the coronavirus and its extended effects continue to throw 
curveballs—such as school re-openings followed by re-closures, 
the tragic string of events leading to the Black Lives Matter 
protests and, perhaps most commonly, the day-to-day pressures 
of living in these economically challenging, socially distanced, 
and life-threatening times—it is critical that the global research 
community do better at capitalizing on what has been learned and 
coordinating diverse networks to determine design and research 
agendas. Institutes of higher education, government agencies, 
industry, and philanthropic organizations must continue to 
support this work by funding high priority research, organizing 
and hosting meetings, building networks and tools and, finally, 
investing in robust broadband, hardware, and training systems. 


The NSF-funded remote methods workshop occurred early in 
the summer of 2020, just months after the coronavirus was 
declared a pandemic. We have learned much since then as a field, 
and we hope that the recommendations in this report are useful 
in inspiring next steps toward preparing for a more equitable 
educational future for all families. 
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Appendix 


More about the Stanford study 

Over 1,000 scouts applied to the study from dscout’s participant 
pool, who are roughly representative of the U.S. cellphone 
owning population. For parents that fit our criteria (at least one 
5-1o-year-old child whose school building closed due to COVID-19), 
we assigned household income levels ($0-49K, $50-99K, over 
$100K) and randomly selected 37 from various income groups for 
a total of 111 participants; 109 completed the study. Our sample 
was majority female (76%) with some diversity of race/ethnicity 
(55% White, 16% Black, 15% Latinx, 9% Asian, and 4% Middle 
Eastern/North African) and level of education (32% were 

high school graduates while 29% reported post-graduate work 
beyond college). 


Data collection was organized in five parts that participants 
completed over the course of two weeks as outlined in Table 6. 
Each part included multiple-choice survey items, open-ended 
text responses, image uploads, and video prompts. Questions 
were directed to parents and other home caregivers such as 
grandparents or live-in partners. 


We used a variety of analytical methods to explore the data, 


including descriptive quantitative summaries and construction 
of indices reflecting breadth of challenges, diversity and source 
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of resources, and perceived learning outcomes; coding open-ended 
text responses and video transcripts across cases to capture 
variation in themes using grounded theory and deductive 
approaches; developing case portraits to help theorize parent- 
identified examples of learning collected in the diary entries. 


TABLE 6: Mission components and data collected 


Study component Topics covered 

Application Demographics; school status during pandemic; challenges 
with remote learning 

Part 1 Pre-pandemic home and school academic access 
and support 

Part 2 During-pandemic learning resources provided by schools; 
how parents were supplementing and supporting learning 

Part 3 Photo and verbal description of a learning moment; 
identification of activity origin and content; ratings of 
enjoyment and learning (this part required 6 unique 
entries, one per day) 

Part 4 How families were learning about COVID-19; examples of 
questions children were asking and of a conversation they 
had with their child 

Part 5 Reflection on possible benefits of remote learning and 


insights about what and how their child learned; evaluation 
of how well their child kept up with learning, interests, and 
social-emotional wellbeing as well as their own capacity 
to adapt to remote learning 
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TABLE 7: Analytic foci for open-ended responses 


and diary entries 


Focus domains 


Categories 


Learning moments 
documented in 
diary entries 


Types of learning activities; caregiver learning partner 
practices; caregivers’ purposes for the creation or 
extension of learning activities; powerful practices/ 
creative innovations; forms of digital, analog, and 
tangible resources leveraged; parent and child affect; 
child-initiated learning activities 


Challenges related 
to learning 


Child engagement and understanding; family coordination; 
parent roles as guides and teacher; parent and child 

social and emotional wellbeing; teacher/school support; 
technological troubles 


School supports 


Parents’ needs; perceptions of valuable resources; 


for learning desires for more support 

Benefits Sources of parent insights about learning; family bonding; 
closer parent-teacher relationships; new skills 

Hopes Hopes for child’s academic learning and social wellbeing; 


Learning about 
COVID-19 
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family relationship; value systems 


Resources for learning; children’s questions; conversational 
content; STEM topics explored; parents’ roles 


More about the University of Washington study 


TABLE 8: University of Washington study participant 


demographics (continues on page 36) 


Group Participant ID Race/ethnicity Children ages Household 
in household in household income 
A PLA African American, 3,10 < $10k 
Asian 
P2_A White iS) $50-100k 
P3_A Asian-Pacific Pd, ale}, aly > $150k 
Islander 
P4_A White, Latino 3,6 $50-100k 
P5_A African American 9,9 > $150k 
P6_A White 4,10 > $150k 
P7_A White, Asian- i $100-150k 
Pacific Islander 
P8_A White Hispanic, 8,5 $50k-$100k 
Asian Pacific 
Islander 
PIA White, African 10 $50-100K 
American 
P10_A White 2,4, 6, 8,9 N/A 
P11A White, Latino <1,2,5,9 §10k-§50k 
P32_A White Sus), Laid off 
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TABLE 8: University of Washington study participant 


demographics (continued from page 35) 


Group Participant ID Race/ethnicity Children ages Household 
in household in household income 
B P12_B White 9 $50k-$100k 
P13_B Hispanic or N/A $50k-$100k 
Latino, White 
P14_B White, Asian / TOR. $50k-$100k 
Pacific Islander 
P15_B White, Black or N/A $50k-$100k 
African American 
P16_B White aki 7/ $10k-$50k 
P17_B Black or African 12, 16 $50k-$100k 
American 
P18_B White 11 $10k-$50k 
P19_B Sri-Lankan N/A $100k-$150k 
and Italian 
P31_B White, Middle 6 $100k-$150k 
Eastern 
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Group Participant ID Race/ethnicity Children ages Household 
in household in household income 
G R208€ White 3,4 $10k-S50k 
Pel White Psy leyehy Urs $10k-S50k 
P2206 White 4 $10k-$50k 
P23_C Black or African 3 $50k-$100k 
American 
P24_C White sh 8) $10k-S50k 
B25u€ White 1,4,6 $10k-S50k 
P26_C White 13, 16, 18 $10k-$50k 
P27_C Arab 5 $50k-$100k 
P28_C White 7,8, 10 < $10k 
P29_C White, Asian / 3 $50k-$100k 
Pacific Islander 
P30_C White all $10k-S50k 
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More about the University of Michigan study 


TABLE 9: University of Michigan study participants and 


TABLE 8: University of Washington study participant 
contexts 


demographics (continues on page 36) 


Great Lakes Great Lakes City West Coast City West Coast City Focus domains Categories and examples 
Community Center public schools Design Center Public Schools 
What information is Form: Narratives, graphics, simulations, scientific 
14 adults (parents/guardians of youth, 6 adults (parents/guardians of youth, accessed explanations, images, etc. 
school and community educators, school and community educators, Focus: Evaluating information: How participants 
district leaders or community-based district leaders or CBO providers) decide what is trustworthy and why. How this 
organization providers) decision relates to racialized, politicized, and 
ethical stances. 
14 youths across Boys and Girls Club, 6 youths across WCCDC and public 
Lansing and public schools (12-18) schools (12-18) How/Where/Who Social media, news, family members, school/district 
information is accessed communication, etc. 
5 young adults (ages 19-29) 17 young adults (ages 19-29) How these forms of information are accessed through 


social networks. 


When information is Early/mid/late in pandemic timeline. 

accessed 

Why information is Knowledge-building/interest, personal decision-mak- 
accessed & how ing, health decision-making, action-taking, etc. 
contextualized How knowledge-building, etc. relates to/is complicat- 


ed by racialized, politicized, and ethical stances. 


Analysis How information is combined and evaluated across 
political, ethical, scientific, and public health sources. 


Conclusions and Decisions that are made, revisited, or revised. 
implications 

Events that stand out Personal stories about oneself and loved ones who 
and why contract COVID-19, news events related to federal 


recommendations, reactions to anti-quarantine 
protests, etc. 
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